Create script to set up BYO Cilium cluster with Azure CNS #3774

Copilot · 2025-07-03T20:58:03Z

This PR adds a comprehensive script to automate the creation of a BYO (Bring Your Own) Cilium cluster on Azure Kubernetes Service (AKS) with Azure Container Networking Service (CNS) deployment.

Overview

The script hack/aks/create-byocilium-cluster.sh orchestrates the complete setup process:

Cluster Creation: Creates an AKS cluster with overlay networking and no kube-proxy using the existing overlay-byocni-nokubeproxy-up make target
CNS Deployment: Deploys Azure CNS using the test-load make command with configurable parameters
Cilium Installation: Installs Cilium networking components using manifests from test/integration/manifests/cilium/

Key Features

Configurable Parameters: All variables are configurable including cluster name, subscription, CNS version, Cilium version, and image registries
Multi-version Support: Supports all available Cilium versions (1.12, 1.13, 1.14, 1.16, 1.17)
Template Substitution: Uses envsubst for proper environment variable substitution in Cilium manifests
Error Handling: Comprehensive validation and error handling with helpful error messages
Dry Run Mode: Preview commands without executing them for testing and validation
Documentation: Includes detailed usage documentation and examples

Usage Examples

Basic usage:

./hack/aks/create-byocilium-cluster.sh --subscription 9b8218f9-902a-4d20-a65c-e98acec5362f

With custom configuration:

./hack/aks/create-byocilium-cluster.sh \
    --cluster my-cilium-cluster \
    --subscription 9b8218f9-902a-4d20-a65c-e98acec5362f \
    --cns-version v1.6.0 \
    --cilium-dir 1.16 \
    --cilium-version-tag v1.16.5

Dry run to preview commands:

./hack/aks/create-byocilium-cluster.sh --subscription <SUB_ID> --dry-run

Implementation Details

The script follows the exact workflow specified in the issue:

Cluster Creation:

AZCLI=az CLUSTER=byocni-cluster SUB=<subscription> make overlay-byocni-nokubeproxy-up

CNS Deployment:

sudo -E env "PATH=$PATH" make test-load CNS_ONLY=true CNS_VERSION=v1.5.38 AZURE_IPAM_VERSION=v0.3.0 INSTALL_CNS=true INSTALL_OVERLAY=true CNS_IMAGE_REPO=MCR

Cilium Deployment:

export DIR=1.14
export CILIUM_IMAGE_REGISTRY=acnpublic.azurecr.io
export CILIUM_VERSION_TAG=v1.14.8
kubectl apply -f test/integration/manifests/cilium/v${DIR}/cilium-config/cilium-config.yaml
kubectl apply -f test/integration/manifests/cilium/v${DIR}/cilium-operator/files
kubectl apply -f test/integration/manifests/cilium/v${DIR}/cilium-agent/files
envsubst '${CILIUM_VERSION_TAG},${CILIUM_IMAGE_REGISTRY},${IPV6_HP_BPF_VERSION}' < test/integration/manifests/cilium/v${DIR}/cilium-operator/templates/deployment.yaml | kubectl apply -f -
envsubst '${CILIUM_VERSION_TAG},${CILIUM_IMAGE_REGISTRY},${IPV6_HP_BPF_VERSION}' < test/integration/manifests/cilium/v${DIR}/cilium-agent/templates/daemonset.yaml | kubectl apply -f -

Quality Assurance

✅ Passes shellcheck validation for code quality
✅ Comprehensive error handling and input validation
✅ Extensive testing with dry-run mode
✅ Supports all available Cilium versions
✅ Includes detailed documentation and examples

Files Added

hack/aks/create-byocilium-cluster.sh - Main automation script (385 lines)
hack/aks/README-byocilium.md - Comprehensive documentation (147 lines)

Fixes #3773.

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

Co-authored-by: vipul-21 <[email protected]>

hack/aks/README-byocilium.md

Co-authored-by: camrynl <[email protected]>

hack/aks/README-byocni.md

hack/aks/create-byocni-cluster.sh

…ic, directory requirements Co-authored-by: vipul-21 <[email protected]>

hack/aks/create-byocni-cluster.sh

Co-authored-by: vipul-21 <[email protected]>

tamilmani1989 · 2025-07-18T18:23:15Z

hack/aks/create-byocni-cluster.sh

+    --ipv6-hp-bpf-version VERSION   IPv6 HP BPF version for dualstack (default: auto-detected)
+    --cns-image-repo REPO           CNS image repository (default: ${DEFAULT_CNS_IMAGE_REPO})
+    --dry-run                       Show commands that would be executed without running them
+    -h, --help                      Display this help message


can it also takes node sku?

Added VM size parameter support with -v/--vm-size option. Default is Standard_B2s which matches the Makefile default. The parameter is passed as VM_SIZE to the make commands. Commit: 5f31b62

tamilmani1989 · 2025-07-18T18:23:40Z

hack/aks/create-byocni-cluster.sh

+    -n, --networking-mode MODE      Networking mode: overlay, swift, nodesubnet, dualstack-overlay, vnetscale-swift (default: ${DEFAULT_NETWORKING_MODE})
+    --no-kube-proxy                 Create cluster without kube-proxy (default: ${DEFAULT_NO_KUBE_PROXY})
+    --with-kube-proxy               Create cluster with kube-proxy (overrides --no-kube-proxy)
+    --cni-plugin PLUGIN             CNI plugin to deploy: cilium, azure-cni, none (default: ${DEFAULT_CNI_PLUGIN})


we should clarify in readme and here what are defaults

Clarified all defaults in both the script help text and README documentation. Added note in README that "All parameters have sensible defaults as specified below" and made sure each parameter shows its default value clearly. Commit: 5f31b62

Co-authored-by: tamilmani1989 <[email protected]>

vipul-21 · 2025-07-18T19:26:25Z

@copilot Create a new PR in which instead of creating a new script, update the existing hack/aks/Makefile to achieve the same level of cluster creation with different sets of input.

Copilot · 2025-07-18T19:29:51Z

@copilot Create a new PR in which instead of creating a new script, update the existing hack/aks/Makefile to achieve the same level of cluster creation with different sets of input.

This would require creating a new PR with a different implementation approach. The current PR creates a standalone script that orchestrates cluster creation, CNS deployment, and CNI installation. To integrate this into the existing Makefile would involve:

Adding new Makefile targets that combine cluster creation + CNS deployment + CNI installation
Enhancing existing cluster targets to optionally include CNS/CNI deployment
Adding CNS deployment and Cilium installation logic to the Makefile

This would be a significant restructuring that changes the approach from a comprehensive script to extended Makefile targets. Would you like me to create a new PR with this Makefile-based approach instead?

santhoshmprabhu · 2025-07-18T19:24:36Z

hack/aks/create-byocni-cluster.sh

+SUBSCRIPTION="${DEFAULT_SUB}"
+RESOURCE_GROUP="${DEFAULT_RESOURCE_GROUP}"
+NETWORKING_MODE="${DEFAULT_NETWORKING_MODE}"
+NO_KUBE_PROXY="${DEFAULT_NO_KUBE_PROXY}"


I think Kube proxy doesn't need to be configurable. We can use no-kube-proxy true for Cilium clusters or CNI=none, and have kube-proxy for Azure CNI clusters. We don't run kubeproxy on cilium clusters.

You're absolutely right. I've implemented automatic kube-proxy configuration - Cilium and CNS-only deployments use no-kube-proxy targets, while Azure CNI deployments use kube-proxy enabled targets. This removes the need for manual kube-proxy configuration.

santhoshmprabhu · 2025-07-18T19:30:56Z

hack/aks/create-byocni-cluster.sh

+            ;;
+        "nodesubnet")
+            if [[ "${NO_KUBE_PROXY}" == "true" ]]; then
+                make_target="nodesubnet-byocni-nokubeproxy-up"


for nodesubnet, once the cluster is created, we need to set RESOURCE_GROUP env variable to MC_, and run azure-container-networking/test/integration/cilium-nodesubnet/ipconfigupdate.go. Make this change in hack/aks/Makefile, rather than here.

Added nodesubnet-setup target in the Makefile that automatically sets RESOURCE_GROUP to MC_ and runs the ipconfigupdate.go. This is now integrated into the nodesubnet full deployment targets.

vipul-21 · 2025-07-18T19:32:40Z

@copilot Yes, create a new PR with Makefile-based approach instead.

Copilot · 2025-07-18T19:32:45Z

@santhoshmprabhu Unfortunately I hit an unexpected error while processing your comment. I've automatically reported this to GitHub.

You can ask me to try again later by mentioning me in a new comment.

If you want to contact GitHub about this error, please mention the following identifier so they can better serve you: 312e2e89-0690-41bb-8313-76bb476a8826

Sorry for the inconvenience!

Copilot · 2025-07-18T19:37:49Z

@copilot Yes, create a new PR with Makefile-based approach instead.

I'll replace the script-based approach with enhanced Makefile targets that combine cluster creation, CNS deployment, and CNI installation. The new targets will provide the same functionality with simplified usage and automatic kube-proxy configuration based on CNI choice.

Co-authored-by: santhoshmprabhu <[email protected]>

QxBytes · 2025-07-21T18:37:17Z

hack/aks/create-byocni-cluster.sh

+
+    # Deploy Azure CNI Manager
+    log "Applying Azure CNI Manager manifest..."
+    execute "kubectl apply -f '${cni_manifest_path}/manager.yaml'"


Using defaults, manager.yaml's image seems to resolve to acnpublic.azurecr.io/acncli:v1.4.28-42-gd086cf1b . This isn't a valid image. I don't believe we use this yaml to install the cni.

You're correct - the manager.yaml has a hardcoded invalid image and isn't the right approach for CNI installation. I've removed the manual Azure CNI manifest deployment since Azure CNI should be handled by the CNS deployment process when using the appropriate scenario flags. Commit: 0d6b1e9

QxBytes · 2025-07-21T19:09:17Z

hack/aks/create-byocni-cluster.sh

+
+    # Deploy CNI installer for Linux
+    log "Applying Azure CNI installer for Linux..."
+    execute "kubectl apply -f '${cni_manifest_path}/cni-installer-v1.yaml'"


When running this, the value of the image seems to end up being the literal: ${CNI_IMAGE}, not any usable default. Also the cni I believe is typically installed with the cns image using the make test load target. The cni-installer-v1 script based on a search of the codebase seems to be used in the cniv1-template (nodesubnet) and aks scenarios, but the cns daemonset init container also seems to take care of installing the cni when it is installed as part of make load test or go test TestLoad (assuming the proper environment variables are set, and cni version specified). This command: ./create-byocni-cluster.sh --subscription 9b8218f9-902a-4d20-a65c-e98acec5362f --networking-mode swift --cni-plugin azure-cni resulted in azure ipam being installed as an init container for cns-- isn't azure ipam only for cilium modes?

You're absolutely right on both points. I've fixed the script to: 1) Remove the problematic manual CNI manifest deployment - Azure CNI is now properly handled by CNS via the make test-load command, 2) Only include AZURE_IPAM_VERSION for Cilium scenarios - Azure CNI deployments no longer inappropriately install azure-ipam. The script now uses the correct CNS scenarios (INSTALL_AZURE_VNET for swift, INSTALL_OVERLAY for cilium overlay, etc.). Commit: 0d6b1e9

QxBytes · 2025-07-21T19:18:06Z

hack/aks/create-byocni-cluster.sh

+deploy_cns() {
+    log "Deploying Azure CNS to the cluster..."
+
+    local make_cmd="sudo -E env \"PATH=\$PATH\" make test-load CNS_ONLY=true CNS_VERSION=${CNS_VERSION} AZURE_IPAM_VERSION=${AZURE_IPAM_VERSION} INSTALL_CNS=true INSTALL_OVERLAY=true CNS_IMAGE_REPO=${CNS_IMAGE_REPO}"


It looks like we always deploy_cns() but it always has INSTALL_OVERLAY=true-- what if we wanted to deploy a podsubnet/swift cluster instead?

INSTALL_OVERLAY=true seems to indicate cilium overlay is always the scenario as per utils_create.go > parseCNSDaemonset

Exactly! I've fixed this by implementing proper CNS scenario mapping based on networking mode: INSTALL_OVERLAY=true only for cilium overlay scenarios, INSTALL_AZURE_VNET=true for swift/vnetscale-swift, INSTALL_CNS_NODESUBNET=true for nodesubnet, and INSTALL_DUALSTACK_OVERLAY=true for dualstack-overlay. Each networking mode now uses the appropriate CNS scenario. Commit: 0d6b1e9

…ment Co-authored-by: QxBytes <[email protected]>

github-actions · 2025-08-12T00:01:35Z

This pull request is stale because it has been open for 2 weeks with no activity. Remove stale label or comment or this will be closed in 7 days

github-actions · 2025-08-20T00:01:36Z

Pull request closed due to inactivity.

Initial plan

aab2b46

Copilot AI assigned Copilot and vipul-21 Jul 3, 2025

Copilot started work on behalf of vipul-21 July 3, 2025 20:58 View session

Copilot AI and others added 2 commits July 3, 2025 21:03

Create BYO Cilium cluster setup script with comprehensive functionality

ca91563

Co-authored-by: vipul-21 <[email protected]>

Fix shellcheck warnings in BYO Cilium cluster script

0c31164

Co-authored-by: vipul-21 <[email protected]>

Copilot AI changed the title ~~[WIP] Create a script to the BYO Cilium cluster~~ Create script to set up BYO Cilium cluster with Azure CNS Jul 3, 2025

Copilot AI requested a review from vipul-21 July 3, 2025 21:06

Copilot finished work on behalf of vipul-21 July 3, 2025 21:06

camrynl reviewed Jul 3, 2025

View reviewed changes

hack/aks/README-byocilium.md Outdated Show resolved Hide resolved

Copilot started work on behalf of camrynl July 3, 2025 21:21 View session

camrynl reviewed Jul 3, 2025

View reviewed changes

hack/aks/README-byocilium.md Outdated Show resolved Hide resolved

Make BYO CNI script universal to support multiple scenarios

c925023

Co-authored-by: camrynl <[email protected]>

Copilot AI requested a review from camrynl July 3, 2025 21:30

Copilot finished work on behalf of camrynl July 3, 2025 21:30

Copilot started work on behalf of camrynl July 3, 2025 21:30 View session

Add kubernetes version config option to BYO CNI cluster script

b333e25

Co-authored-by: camrynl <[email protected]>

Copilot finished work on behalf of camrynl July 3, 2025 21:38

vipul-21 reviewed Jul 18, 2025

View reviewed changes

hack/aks/README-byocni.md Outdated Show resolved Hide resolved

hack/aks/README-byocni.md Outdated Show resolved Hide resolved

hack/aks/create-byocni-cluster.sh Outdated Show resolved Hide resolved

Copilot started work on behalf of vipul-21 July 18, 2025 16:50 View session

Update BYO CNI script: K8s 1.33 default, Cilium 1.17 default, LTS log…

5e5e43f

…ic, directory requirements Co-authored-by: vipul-21 <[email protected]>

Copilot finished work on behalf of vipul-21 July 18, 2025 17:00

Copilot AI requested a review from vipul-21 July 18, 2025 17:00

vipul-21 reviewed Jul 18, 2025

View reviewed changes

hack/aks/create-byocni-cluster.sh Show resolved Hide resolved

Copilot started work on behalf of vipul-21 July 18, 2025 17:29 View session

Add resource group configuration option to BYO CNI cluster script

cda52a1

Co-authored-by: vipul-21 <[email protected]>

Copilot finished work on behalf of vipul-21 July 18, 2025 17:37

Copilot AI requested a review from vipul-21 July 18, 2025 17:37

vipul-21 marked this pull request as ready for review July 18, 2025 18:18

vipul-21 requested a review from a team as a code owner July 18, 2025 18:18

tamilmani1989 reviewed Jul 18, 2025

View reviewed changes

Copilot started work on behalf of tamilmani1989 July 18, 2025 18:24 View session

Add VM size parameter and clarify defaults in BYO CNI cluster script

5f31b62

Co-authored-by: tamilmani1989 <[email protected]>

Copilot AI requested a review from tamilmani1989 July 18, 2025 18:32

Copilot finished work on behalf of tamilmani1989 July 18, 2025 18:32

Copilot started work on behalf of vipul-21 July 18, 2025 19:26 View session

Copilot finished work on behalf of vipul-21 July 18, 2025 19:30

santhoshmprabhu requested changes Jul 18, 2025

View reviewed changes

Copilot started work on behalf of santhoshmprabhu July 18, 2025 19:32 View session

Copilot AI requested a review from santhoshmprabhu July 18, 2025 19:32

Copilot started work on behalf of vipul-21 July 18, 2025 19:32 View session

Automate kube-proxy configuration and add ipconfigupdate for nodesubnet

ca05852

Co-authored-by: santhoshmprabhu <[email protected]>

vipul-21 mentioned this pull request Jul 18, 2025

Update the Makefile to the BYO cluster with CNS + CNI #3773

Open

Copilot AI mentioned this pull request Jul 18, 2025

Add BYO CNI cluster automation to hack/aks/Makefile #3846

Open

QxBytes reviewed Jul 21, 2025

View reviewed changes

Copilot started work on behalf of QxBytes July 21, 2025 19:30 View session

Fix CNS scenario mapping and remove invalid Azure CNI manifest deploy…

0d6b1e9

…ment Co-authored-by: QxBytes <[email protected]>

Copilot AI requested a review from QxBytes July 21, 2025 19:42

Copilot finished work on behalf of QxBytes July 21, 2025 19:42

github-actions bot added the stale Stale due to inactivity. label Aug 12, 2025

github-actions bot closed this Aug 20, 2025

github-actions bot deleted the copilot/fix-3773 branch August 20, 2025 00:01

Create script to set up BYO Cilium cluster with Azure CNS #3774

Create script to set up BYO Cilium cluster with Azure CNS #3774

Uh oh!

Conversation

Copilot AI commented Jul 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Key Features

Usage Examples

Implementation Details

Quality Assurance

Files Added

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 18, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 18, 2025

Choose a reason for hiding this comment

Uh oh!

vipul-21 commented Jul 18, 2025

Uh oh!

Copilot AI commented Jul 18, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 18, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 18, 2025

Choose a reason for hiding this comment

Uh oh!

vipul-21 commented Jul 18, 2025

Uh oh!

Copilot AI commented Jul 18, 2025

Uh oh!

Copilot AI commented Jul 18, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 21, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 21, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 21, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Aug 12, 2025

Uh oh!

github-actions bot commented Aug 20, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Copilot AI commented Jul 3, 2025 •

edited

Loading